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Abstract 

In the Bradley- Terry model for paired comparisons, and its extensions to include order effects and 
ties, the maximum likelihood estimates of probabilities of certain outcomes can be or 1 under certain 
data configurations. This poses problems for standard estimation methods. In this paper, we give 
algorithms for identifying the outcomes with estimated probability or 1, and indicate how the remaining 
probabilities may be estimated and summarized. 



1 Introduction 



A paired-comparison experiment is used to assess the relative worths of t objects when they can only be 
compared two at a time, and when the result of such a comparison yields only the result that one of the objects 
was preferred to the other (or, possibly, that the two objects in the comparison were equally preferable). 



The most commonly-used model for data from such an experiment is that of Bradley and Terry (1952) 



with extensions by Davidson (1970) and Davidson and Beaver (1977) These models yield estimates of the 



probability that one object will be preferred to another. Davidson and Farquhar (1976) give an extensive 
bibliography of Bradley- Terry and related models. 

Applications of the Bradley- Terry model include taste-testing, in which each comparison is carried out 
by a judge, and sporting competition, in which the objects are players or teams and comparisons represent 
games between them, with preferences corresponding to wins and losses (or ties). In this paper, we use for 
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definiteness the terms "teams" and "games" to refer to the objects and comparisons, though our results 
apply more widely. 

A drawback of the Bradley- Terry model is that it is possible for estimated probabilities to equal or 
1, and therefore the maximum likelihood estimates of the parameters are not guaranteed to exist (in the 
sense of being in the interior of the parameter space). This was noted by Ford (1957)[ who gave a necessary 



and sufficient condition for the estimated probabilities to lie strictly between and 1 in Bradley and Terry's 
original model. 

The Bradley- Terry model and its extensions can be viewed as special cases of (binomial and trinomial) 



logistic regression models. Albert and Anderson (1984) and Santner and Duffy (1986) give conditions which 



determine whether the maximum likelihood estimates in logistic regression are finite or infinite, and give 
linear programs and other methods for determining the effect of any nonexistent parameter estimates on the 
estimated probabilities. 

In this paper, we use the results of lAlbert and AndersonI and [Santner and Duffy| to show that a simple 
procedure, based on ideas of graph theory, leads to a complete analysis of a paired comparison experiment 
using the original Bradley- Terry model and one of its extensions, even when the maximum likelihood esti- 
mates do not all exist. We also demonstrate that variations on this procedure apply to the other extensions 
of the model. In Section[21 we review the Bradley- Terry model and its extensions; in Section|31we consider 
estimation when the only possible outcomes are wins and losses, and in Section^jwe consider what happens 
when ties are included in the model. Finally, we discuss the presentation of results from our methods in 
Section [SI and give some illustrative examples in Section [^ 

2 Bradley-Terry model and extensions 
2.1 Bradley and Terry's original model 



The Bradley and Terry (1952) model assumes that each team has a "worth" or "strength" tt^ > reflecting 
the i-th team's tendency to win (or lose). The tt^ are determined only up to ratios, and so in practice an 
additional condition, such as ""i = 1 or vrt = 1, is imposed to force the maximum likelihood estimates 

to be well-defined. The Bradley- Terry model asserts that the probability pij of team i defeating team j is 

= „ ; (1) 
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equivalently, T^iji^j represents the odds in favour of i defeating j. (The Bradley- Terry model is therefore a 
"proportional-odds" model.) 

If each pair of teams i and j play n.y games against each other with team i winning of them, and all 
games are assumed independent, the likelihood is seen to be 



Fienberg (1979) shows that the likelihood equations require the observed and expected numbers of wins for 
each team to be equal (where the expected number of wins is the sum of win probabilities over games played 
by the team). 

2.2 An order effect 

In taste testing, an object may be more likely to be preferred simply because it is presented first rather than 
second; in sports, a team is typically more likely to defeat another when playing at home. This "order effect" 
or "home field advantage" can be included in the model by assuming the probability of i defeating j in that 
order (that is, with team i at home) to be 

V^j = ■ (2) 

77ri -I- TTj 

The additional parameter 7 > represents the advantage of being first (playing at home) ; it is assumed 



to be the same for all teams. This model was proposed by Davidson and Beaver (1977) As in Section [TTl 
the likelihood equations equate the observed and expected wins for each team; the additional equation for 7 
equates the observed and expected wins for the home team over all games played. 

2.3 Team-specific order eflFects 

It is often the case that a single home field (order) effect is consistent with the data, though this may 
be because there is insufficient data to prove that a separate home field effect is necessary for each team. 
Nonetheless, it is straightforward to generalize Q to allow team-specific home field effects. This is most 
easily done by observing that we are effectively treating "team i at home" and "team i on the road" as 
completely separate entities. Letting TTi/f denote the strength of team i at home and -Kiy the strength of 
team i on the road (as "visiting team"), the probability of team i defeating team j with i at home can 
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be modelled as 



= - — — ;— (3) 

and the likelihood equations now equate observed and expected home and road wins for each team (a total 
of 2t equations for the 2t parameters, though one equation and one parameter are redundant). 

2.4 Ties and the Bradley- Terry model 

If one of the possible outcomes of a comparison is "no preference" (that is, "tie"), then this should be 
included in the model. One straightforward approach is simply to count a tie as half a win and half a loss 
and use (O or J^J as appropriate. This violates the binomial assumption underlying those models, but 
can work well in practice if the goal is simply to estimate the strengths of the teams (rather than estimating 



probabilities). Davidson (1970) proposed a generalization of the Bradley- Terry model that allows for a tie 
as a third outcome. Letting pijk denote the probability of outcome type fc (fc = 1 if i wins, /c = 2 if j wins, 
fc = if a tie), the generalization is 

Pijl 



PyO - 



The additional parameter v >Q reflects the tendency for ties to happen, and is assumed to be the same 
for all teams. This model has two desirable properties: the conditional probability of i defeating j, given 
a non-tie, is as in and the probability of a tie, for fixed v, is maximum when tt; = tTj. The likelihood 
equation for v equates the observed and expected ties over all games; the remaining equations now equate 
the observed and expected points for each team, where one point is given for a win and half a point for a tie 
(or equivalent such as two points for a win and one for a tie). 



Davidson and Beaver (1977) show how ties and order effects may be modelled simultaneously; it is also 



possible to combine the above model with (jSJ. 
2.5 Team-specific tie effects 

Model assumes that all teams have the same tendency to tie games, and this may not always be reasonable. 



For example, Joe (1990) found that some chess players tend to draw more matches than others, even after 
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allowing for the strength of opposition faced. In this case, one can replace v hy a, Vi for each team, as follows: 









^3 










T^i - 






^3 
















^3 



^'*i2 = _|- TT. -I- ^'/7A-7/,7r.Tr, 

Pija 

with the likelihood equations now equating the observed and expected points and ties (and therefore wins) 
for each team. This model can also be combined with one containing order effects. 

2.6 Numerical methods 



Ford (1957) describes the following procedure for the model Q: the likelihood equations set, for team 



t 



TLj 



and therefore 

t t 

This can be treated as a fixed-point iteration, using the current parameter values on the right-hand side 
to produce an updated value for the parameter on the left. Ford shows that, if the maximum likelihood 
estimates Tr^ all exist, the values from the iterative procedure will converge to the maximum likelihood 
estimates. Convergence is generally slow, but the procedure is reliable and easy to program. The same idea 
can be used in the extensions to the Bradley- Terry model. 



Several other iterative procedures are available, such as those described and compared in Butler (1997, Section 3.3) 
These also assume that the parameter estimates Hi all exist. They tend to require fewer iterations than Ford's 
method, but have greater computational cost per iteration so that an overall improvement in speed is not 
guaranteed. 

However, the existence of the Tr^ is not normally apparent, and if one or more of the tt^ is actually 
infinite or zero, an iterative procedure will only converge slowly, if at all. Indeed, the usual reason for such 
a procedure to reach its iteration limit is that some parameter estimates do not exist. 
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3 Estimation with win/loss outcomes 



3.1 Introduction 

When parameter estimates may not exist, a much more satisfactory approach is first to identify the nonex- 
istent ones (and hence the fitted probabilities that are f or 0), and only then to apply an iterative procedure 



if necessary. Ford (1957) showed that, in the estimates will all exist if and only if there is no division of 
the teams into two groups A and B such that every game between a team in A and a team in B was won by 
the team in A. More generally, in logistic regression, some observations may have fitted probabilities that 
are or 1, and it is desired to isolate and remove these observations before applying an iterative procedure 



to estimate the remainder of the probabilities; this is the problem addressed by Albert and Anderson (1984) 



and Santner and Duffy (1986) 



3.2 Simple Bradley- Terry model 



The model is written as a logistic regression model by letting f3i — logTTj; Albert and Anderson (1984) 
showed that the existence of maximum likelihood estimates in such a model was connected with the existence 
of certain solutions to the corresponding logistic discrimination problem. In the present context, that problem 
is defined by requiring 

ft -P3>0 (6) 

if there exists at least one game in which i has defeated j. 

lAlbert and AndersonI and [Santner and Duffy] then characterize the cases in which maximum likelihood 
estimates fail to exist by defining "complete separation" , "quasi-complete separation" and "overlap" . For 
complete separation, there must exist ft satisfying ^ with strict inequality for all observations. In this 
case, all the fitted probabilities (for outcomes which actually occurred) and the maximized likelihood itself 
are 1. For quasi-complete separation, there must exist ft satisfying jnj with at least one strict inequality 
and at least one equality. In this case, the maximum of the likelihood is less than 1, but some of the fitted 
probabilities are 1. Overlap occurs otherwise (that is, when the only solution to JBJ is with equality and 
all the ft are equal); in this case, it is shown that all the maximum likelihood estimates all exist and all 
the estimated probabilities lie strictly between and 1. (Note that the ft in a sense play a dual role: for 
determining the separation properties of the data, we are concerned only with the existence of non-equal ft 
solving the discrimination problem; the ft in the likelihood itself for our regression problem represent the 
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relative strengths of the teams, and the values will be different in the two contexts. For clarity, we will, as 
much as possible, work with (3 when discussing the question of separation and tt when dealing with maximum 
likelihood estimates.) 

For any two teams i and j, define the relation > by i > j if there is a game in which i has defeated j. 
(Note that we can have i> j and j > i, if for instance i and j have played twice and won one game each.) 
For consistency in the sequel, we also define i > i to he true for each team i. 

Define the relation ^hy i > j if i > j or if there exist teams fci, /c2, . . . /c^ such that i > ki > ■ ■ ■ > kr > j . 
Now consider the t teams as vertices of a directed graph G which has an edge from vertex i to vertex j if 
i > j. Determining whether the relation !> holds for a pair of teams k and I is equivalent to determining 
whether there is a path in the directed graph from the vertex representing k to that representing I - that 
is, whether there is an edge from k to I in the transitive closure G of G. The transitive closure of a 
graph can be computed in 0{t^) time by an algorithm due to Floyd and Warshall - see, for example, 
Papadimitriou and Steiglitz (1998, p. 132) 

For each pair of teams i and j, i \> j ov not and j \> i ov not. Define the relations ^, 3>; ^ as follows: 

i i if i C> j and j !> i 

i ^ j if i ^ j only 

i ^ j if neither « !> j nor j > i 

It is possible, though redundant, to define z ^ j if and only if j 3> i. Further, i ~ i since i > i and hence 
i > i. 

It is easily seen that = is an equivalence relation and that 3> is an ordering. We assert that when i ^ j, 
the fitted probability of i defeating j lies strictly between and 1, when i ^ j, the fitted probability is 1, 
and when i ^ j, the fitted probability is arbitrary since there is no basis for comparison between i and j. 

To apply the Albert- Anderson-Santner-Duffy theory, we assign Pi such that Pi = Pj if i ^ j. For i ^ j, 
we assign values such that Pi > Pj . These assignments are possible since = is an equivalence relation and 3> 
is an ordering. Provided that all teams have played at least one game, this permits values to be assigned to 
all Pi. In each case, the numerical values of the Pi are arbitrary; it is only the ordering that is of importance. 

To show that this correctly identifies complete separation, quasi-complete separation, and overlap, note 
first that i ^ j if and only if i !> j and j ^ i. Thus there exist (possibly empty) strings of teams such 
that i > ki > ■ ■ ■ > kr > j and j > h > ■ ■ ■ > h ^ i- The discrimination equation (|SJ) implies that 
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Pi > Pki > • ■ ■ > and Pj > Pi^ > ■ ■ ■ > Pi- This can only be satisfied by Pi = Pj. Thus Pi — Pj if and 
only if i ^ j. When i ^ j, however, Pi > Pj; at least one of the inequalities in the chain must be strict since 
otherwise j t> i as well. Finally, if i ^ j, neither i = j nor j = i, which means that no chain of inequalities 
between Pi and Pj in either direction can be inferred from the discrimination equation. Thus there exist 
solutions with either sign of Pi — Pj. (This is only possible if i and j did not play one another, and thus is 
only relevant to the probabilities of outcomes of games which did not occur.) It follows that if no pair of 
teams is related by ~, the Pi may all be chosen to be different, and we have complete separation. If there 
exist teams i and j with i ^ j and also teams k and I with k ^ I (or k ^ I), we take Pi = Pj and Pk > Pi, 
indicating quasi-complete separation. Finally, if all teams are related by =, then Pi = Pj for all and so 
we have overlap. 

The identification of outcomes with or 1 probabilities allows us to replace the probability of a win for 
i in a game against j in |^ with 

: II « = J 



Pij = < 



'3 

1 if i > j 



if j > i 
arbitrary if i it j- 

An "arbitrary" probability is one whose value is not determined by the data (i.e., there exist maximum 
likelihood solutions with all values, between and including and 1, for the probability in question). The 
effect of the modified definition of probability on the maximum likelihood equations is to remove from the 
calculation any games whose outcomes have maximum likelihood probabilities of or 1. The resulting 
strength estimates TTi are all guaranteed to exist and can be found by methods such as those discussed in 
Section 12.51 Maximum likelihood estimates of unit (or vanishing) probability are now associated with a 3> 
relationship between teams, rather than an infinite (or zero) ratio of strength parameters. 

3.3 Team-specific order effects 

In Section f2.3l it is noted that team-specific home field advantage parameters can be modelled by considering 
each team's home performances independently of its road performances. The separation properties of the 
data can therefore be determined exactly as in Section 13.21 with 2t items, the home and road versions of 
teams 1, 2, . . . , 
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3.4 Single order effect 

If in lI2Jl we let [3i — logTT; and 77 = log 7, the Albert- Anderson discrimination condition for a game between 
i and j with i at home and winning (losing) is 

A-/3,+r/> (<)o. 

If we reparameterize, defining (3ih = Pi + ?7/2 and f3iv = f3i — 7^/2, the condition becomes 

P^H - HjV > (<)0. (7) 

(H represents "home team" and V "visiting team".) This is the condition of Section rOl but since we have 
overparameterized, we also have that 

PiH-Piv^---^ fitH - (5tv. (8) 

since each difference is equal to 77. In inferring t> relationships between the teams, we therefore need to take 
into account ((SJ as well as ((TJ. Equation (jS)) has two implications: 

• If there exists a set of teams {fci,...,fcr}, with r > 2, such that Pu^h > Pk-iVi ■■■ Pk^-iH > 
PkrV, PkrH > PkiV, then Pih > Piv for all teams I, since 77 > 0. (The difference between the 
sum of the left-hand sides of the inequalities and the sum of the right-hand sides is rrj.) The same 
applies with H and V interchanged. 

• If (3iH > PjH for any teams then Piv > Pjv for those teams and conversely, since Pi > Pj. 

The procedure of Section rOl is therefore no longer sufficient. An iterative scheme will be necessary in 
its place, since knowledge that 7/ > (<)0 provides additional information about whether Pm > {l£)Piv, and 
conversely. Note, however, that if we once determine that 77 > 0, we do not need to check again (since the 
implications for P^jf and Piv are fixed); the same applies separately when > 77. 

To determine whether 77 > 0, we need to find a chain of inequalities, as described above, beginning and 
ending at the same team. Unless we already have that P^h > Pkv for some k, this is most easily done by 
constructing a directed graph Gh which has vertices representing the teams {1, 2, . . . , t} and edges from i to 
J whenever Pin > Pjv (that is, when iH [> jV). In Gh, no vertex k is considered connected to itself, since 
we only construct Gh if kH !> kV does not hold for any k. We then compute the transitive closure Gh of 
Gh, and determine whether it contains any edges from a vertex k to itself, in which case PkH ^ Pkv and 
hence t] > 0. 
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To determine whether ?7 < 0, the same technique can be apphed with H and V interchanged. 
This leads to the following algorithm: 

1. IH \> IH and IV > IV for each team I (for completeness). 

2. For each game, let i be the home team and j the visiting team. If i defeated j, iH > jV; otherwise, 
jV > iH. 

3. Construct a directed graph G with the 2t vertices IH, .... tH, IV, . . . ,tV and edges connecting those 
vertices related by [>. Compute the transitive closure G of G; any edges {k,l) in G but not G imply 
an additional > relationship between items k and /. 

4. If kH l> kV for any k, set kH l> kV for all k, and go to Step|31 Otherwise, construct a directed graph 
Gh with vertices 1,2, ... ,t and edges from i to j whenever iH > jV. Compute its transitive closure 
Gh- If Gh contains an edge from some vertex I to itself, set kH \> kV for all k. 

5. If kV > kH for any k, set kV > kH for all k, and go to Step|H| Otherwise, construct a directed graph 
Gv with vertices 1,2, ... ,t and edges from i to j whenever iV t> jH. Compute its transitive closure 
Gv- If Gv contains an edge from some vertex I to itself, set kV > kH for all k. 

6. For each i and j, if ii/ ^ jH, then > jV, and conversely. 

7. If steps 13 through on the current cycle added any > relationships, go back to step|21 otherwise, stop. 

It is computationally most convenient to maintain the directed graph G as an adjacency matrix, in which 
the [> relationships can be found and updated directly. 

The maximum number of > relationships is 4i^, so the maximum number of iterations of steps |31 through 
Elis bounded. Often, in practice, only one or two iterations are necessary. 

To determine the probabilities whose maximum likelihood estimates are 1, or arbitrary, we find which 
relation, ^,=, ^ applies between each pair of items, where a item is an entity such as iH, a team with a 
venue attached. (Note that even if iH ^ jV, it is perfectly possible to have jH = iV, depending on the 
magnitude of the home field advantage.) Then, instead of using we define the probability of i defeating 
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j with i at home as 



if iH = jV 



1 if iH:$> jV 

if jV > iH 
arbitrary if iH ^ jV. 

Once again, using this modified probabihty formula in the maximum hkehhood equations effectively 
removes games whose outcomes have maximum likelihood probabilities of or 1 from the data set, and 
produces a solution in which 7 and all the Tii exist. 



4 Estimation in the presence of ties 
4.1 Reparameterization 

The discrimination conditions for models containing ties can be written most conveniently if we define 
parameters which cast the two versions of the model (single or team-specific tie parameters) in the same 
notation, by defining 77;+ = yfiuy in the model or 7ri+ = ^JtuvI in (jSJ, so that the relative probabilities 
of a win, tie, or loss for team i in a game against team j are tt^, 7ri_|_7rj_|_, and tTj, respectively. (Note that the 
model with team-specific tie parameters could in fact be defined in this way, with the tt^-)- taking the place 
of the Vi as independent parameters.) For convenience, we also define tt^^ — ■Ki/ni^^ which means that the 
relative probabilities of a win or tie for team i against team j are tt^- and 7rj4_, respectively. 

Albert and Anderson (1984)|show that, if the response variable in a logistic regression has g categories, 



then each observation contributes 5 — 1 conditions on their a. In the presence of ties, therefore, each game 
gives two conditions on the Pi. If we write (3i = logi^i and [3i+ — logTr^^ in either of the models with ties, 
the conditions are 

> 

> 



if i defeats j, and 



if the result is a tie. 
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If we define an additional parameter — Pi — [3i^ — logvTi-, the conditions take on the simpler form 

(9) 

A- > Pj+ 

if i defeats j, and 

'- (10, 

if the result is a tie. 

To determine which results have maximum-likelihood probabilities which are or arbitrary, we need 
to find the exhaustive set of inequalities which apply among the t parameters f3i, . . . , Pt and also among 
the 2t parameters /3i+, . . . , Pt+, Pi-, • ■ ■ , Pt- as a result of the conditions @ and H1U|) . along with auxiliary 
conditions relating Pi, Pi^, and Pi-. These will be different depending on whether the model contains a 
single tie parameter or one for each team. 

4.2 Single tie parameter 

If there is only one tie parameter, the definitions of the parameters imply Pi± — {Pi ± logz^)/2. This means 
that the discrimination conditions can be re-cast without reference to Pi by replacing JHJ with the conditions 

P^± > Pj± 

by adding or subtracting log v from both sides. The auxiliary relationship 

/3i+ - = • • • = Pt+ - Pt- 

is identical to the condition (|HJ) given in Section 13.41 so the procedure determining the complete set of 
inequalities applying to this model is the same as that in the model with a single order effect, even though 
it is being applied in a different context. Specifically: 

1. 1+ t> 1+ and /— > for each team I (for completeness). 

2. For each game in which i defeated j, i+ > j+, i— j — , and i~ [> j+. For each game in which i tied 
j, *+ ^ j~ and j+ t> i—. 

\'M7\ Carry out the corresponding steps from the single-order-parameter algorithm of Section 13.41 with H 
replaced by 4- and by — . 
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Ptjl 




i ^ i 


i > i 


i ^ i 


i— ^ j+ 


1 




U 


arbitrary 


i— = j+ 









arbitrary 


i+ > i- 
















arbitrary 


arbitrary 





arbitrary 



Table 1: Revised definition of Piji, the probability of team i defeating team j 



PijQ 


i+ > j~ 


i+ = j- 


3- > 


i+ ^ 3- 


3 + 
JH 
i— 


> i~ 

- = i— 


1 









arbitrary 
arbitrary 



> J + 












arbitrary 


arbitrary 





arbitrary 



Table 2: Revised definition of Pijo, the probability of a tie between teams i and j 

Once the full set of t> relationships has been determined, they can be used to determine the or ^ 

relationships between the 2t items 1+, . . . , t+, 1 — , . . . , t—. Since i+ is guaranteed by stepElof the algorithm 
to have the same relationship with j+ as i— does with j— (and to allow an analogy with the following 
section), it is convenient to refer to that relationship as holding between i and j. For example, if i+ ^ j+ 
(which means i— S> j~) we say i 3> j. 

From these relationships, we can conclude that the maximum likelihood estimates of certain probabilities 
are 0, 1 or arbitrary. However, with three possible outcomes, we may have one outcome whose estimated 
probability is zero, while the remaining two probabilities are both estimated to be nonzero. Tables ^ and 
121 show, respectively, the maximum-likelihood probabilities of wins and ties taking the relationships into 
account. The maximum likelihood estimates Tr^, TTj, 7ri+, 7rj+ are guaranteed to exist. 

In the Tables, note that the combination of relationships which gives an arbitrary probability for any 
result is only possible if the two teams have not actually played. Note also that in some cases (for example 
i ^ J, i+ 5? j—, i— ^ j+) the ratio of the probabilities of two outcomes can be well-defined and finite, with 
the ratio of either to the probability of the third outcome unconstrained by the data. That is, the conditional 
probabilities of the first and second outcomes, given that the third outcome does not happen, lie strictly 
between and 1. However, in this case we still consider all three probabilities to be arbitrary. 
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If the maximum likelihood estimates are zero for two of the probabilities Pijo,Piji,Pij2, the third proba- 
bility is estimated to be 1, and any such games can simply be removed from the data set before estimating 
the remaining probabilities. However, in games where exactly one outcome is constrained to have zero prob- 
ability, the modified probabilities from Table H and [21 must be used to leave only those outcomes out of 
likelihood maximization. Since the likelihood equations here equate observed and expected points for each 
team (where points are defined as in Section |2.4|I . as well as observed and expected overall ties, this means 
that we must restrict our calculations to those outcomes whose probabilities are known to lie strictly between 
and 1. 

If we make the obvious generalization that jjijk is the number of games between i and j with outcome fc, 
and take pijk from Tables ^ and El the likelihood equations become, for each tt^, 

j j j j 

0<piji<l 0<j3ijo<l 0<piji<l 0<pijo<l 

and for 

Y y^j^ Y ^ijPijo- 

i<j i<j 
0<Pi3i<l 0<Pijl<l 

These equations yield a method analogous to Ford's for finding the maximum likelihood estimates. In the 
first set of equations, a factor can be removed from the right-hand side and isolated, and in the last 
equation the same can be done with v. Then, in each case, a fixed-point iteration can be carried out. 

4.3 Team-specific tie parameters 

In the case of a team-specific tie parameter model, the auxiliary equations describing the overparameteriza- 
tion that exists due to the introduction of the extra t parameters . . . , Pt- are 

These do not lend themselves to a direct simplification of JHl, but allow us to infer some inequalities among 
parameters by adding and subtracting others. For example, if f3i+ > f3j^ and f3i^ > (3j+, we can add the 
two inequalities to infer Pi > (3j . This produces the following algorithm for determining the > relationships 
amongst the teams 1, . . . , t as well as the items 1+, . . . , t+, 1—, . . . , t—: 

1. 1^1,1+^ l+, and I— t> for each team I (for completeness). 

2. For each game in which i defeated j, i C> j and i— > j+. For each game in which i tied j, i+ !> j— 
and j+ [> i~. 
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3. Construct a directed graph G with vertices 1, 2, . . . , f and with edges from i to j ii i \> j. Compute 
the transitive closure G of G. Any edges {k, I) in G but not G imply additional relationships k > I. 
Repeat with the directed graph G+ whose edges are the 2t items 1+, . . . , i+, 1— , . . . , t— . 

4. For each pair of teams i and j: 

• If * i and j+ > i+, then i— t> j— ; 

• If i j and j'+ > , then i+ [> j— ; 

• If i !> j and j— !> i+, then i— t> j+; 

• If i !> i and j— [> i— , then z+ [> j+; 

• If i+ > _;/+ and i— !> j— , then i \> j; 

• If z+ !> j— and i— > j+, then i !> j. 

5. If steps|21or^on the current cycle added any > relationships, go back to step|21 otherwise, stop. 

As in Section [3. 41 it is computationally convenient to maintain G and G+ as adjacency matrices, for ease of 
determination and updating of the !> relationships. 

Once we have obtained the full set of > relationships among the t teams and the 2t team-and-sign 
items, we use them to find the =, 3>, and ^ relationships among teams and between the "plussed" teams 
1+, . . . ,t+ and the "minussed" teams 1 — , . . . , t— . These can then be used to identify the outcomes whose 
maximum-likelihood probabilities vanish and to deduce correctly the remaining estimated probabilities pijk 
according to Tables ^ and 13 Again, this procedure permits a maximum likelihood solution for the remaining 
probabilities with parameter estimates that exist. This time, the likelihood equations are, for each tt^, 



Again, a Ford-like iterative process can be constructed. In the first set of equations, a tt^ can be factored 
out of the right-hand side and isolated, and in the second set, the same can be done with either y/T^ or iTi^. 




0<Pi3l<l 



0<Piji<l 



and for each v, 




J 

0<Pi3l<l 



3 

0<Piji<l 
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5 Presentation of results 



5.1 Introduction 

Our procedures yield maximum likelihood estimates for all the probabilities pij (or pijk in the presence of 
ties) , but these probabilities may be determined from the existence of, say, a ^ relationship rather than from 
finite parameters that represent the strengths of the teams. It is, nonetheless, desirable to have a "strength" 
measure for each team regardless of whether the parameter estimates all exist; for example, we may wish to 
produce a rank ordering of the teams. 

5.2 Simple Bradley- Terry model 

Here, the maximum-likelihood solution is completely specified by a partition of the teams into equivalence 
classes according to the = relation defined in Section the relations {':$> or ^) between classes, and the 
estimated tt^ for each team, defined up to an overall multiplicative factor within each class. This does not 
permit easy comparison of teams in different classes. 

To provide a ranking of the teams even in the presence of ^ relationships, as well as a single number 
with a concrete interpretation, we can define the round-robin winning percentage, or "RRWP" . This is the 
proportion Ri of games a team would be expected to win if they played each other team an equal number of 
times (as in a "balanced schedule" or "round- robin tournament" ) , and is given by the average of that team's 
estimated winning probability against all of the other teams: 

Arbitrary pij are (for definiteness) set equal to ^ in this calculation. This allows the RRWP to be defined 
given any set of results. This definition gives a sensible ordering of the teams, since if > Pji for a pair of 
teams i and j, then (because pik > Pjk for any other team fc), Ri > Rj. If two teams are related by ^, and 
thus the maximum likelihood probability of one defeating the other is arbitrary, the RRWP can be used to 
provide a somewhat arbitrary ordering of them. We do not have enough information to compare such teams 
directly in the Bradley- Terry model, but the RRWP in effect pulls to the middle of the rankings teams with 
many arbitrary pij , those teams being "unknown quantities" . 

In the case where the teams have actually played a balanced schedule, each team's RRWP will be equal 
to its actual winning percentage. This is a manifestation of the familiar result that the Bradley- Terry model 
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reproduces ordering by won-lost record in the case of a balanced schedule ("complete block design" in the 



language of Bradley and Terry (1952) I. However, this definition extends to the case where some teams 
are related by in which case there are maximum likelihood estimates of parameters in the traditional 
Bradley- Terry model that do not exist. 

5.3 Order effects 

In the presence of order effects, either single or team-specific, the definition of RRWP must be generalized to 
include the home field advantage. The quantity Pij is now the predicted probability for i to win a game at 
home against j, and because of the home field advantage 7^ 1 — Pij in general. Note that the "diagonal" 
probability pa is a measure of the strength of the home advantage; with a single order effect, it is simply 
7/(1 -|- 7), while with team-specific order effects, pu measures the difference between team i's performances 
as a home team and as a visiting team. 

A logical generalization of the RRWP to a model with order effects is the percentage of games a team 
would be expected to win if they played each other team once at home and once on the road: 

R^ - -11 ^^P'' + ^ ~ P''^' ^^^^ 

Once again, arbitrary pij are replaced in the sum by 1/2, and if the teams have actually played this kind 
of round-robin, each team's RRWP will be equal to its actual winning percentage. In a model with a single 
order effect where the home advantage parameter exists {kH = kV for all fc), Ri > Rj if « 3> j, or if i ^ j 
and TTi > TTj. 

5.4 Models with ties 

When ties are allowed in the model, the concept of RRWP must be generalized. Our procedures now give 
estimated probabilities pijk of each result (win, loss or tie) in games between each pair of teams. We can use 
the Pijk to obtain the predicted number of wins, losses, and ties for each team in a round-robin schedule. 
The predicted round-robin percentage Rik of outcome k for team i is 

As in Sections 12.41 and 12.51 k = 1 denotes a win for team i, k = 2 sl win for team j, and k — Q a, tie. All 
arbitrary probabilities for a particular hypothetical (unplayed) game are assigned equal weight, so that if 
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all three probabilities (win, loss, or tie) against a particular opponent are arbitrary, they are all assumed to 
be 1/3 in the sum, while if one probability pijk vanishes, the other two are assumed to be 1/2. One could 
of course define a more sophisticated method of making this assignment, taking into consideration known 
ratios of unknown probabilities, or the overall proportion of ties, but we only present the simplest method 
here. 

These round-robin proportions can be used to define a RRWP, according to the traditional definition of 
a tie as a half a win and a half a loss, by 

Ri = Rii + -R'iQ. (13) 

In the single tie parameter model, if piji > Pij2, then Ri > Rj. This means that if no teams are related by 
^ the ranking of the teams by strength and by RRWP will be identical. This is not true in the model with 
team-specific tie parameters, as illustrated in Section 16.31 

The round-robin proportions predicted by the team-specific tie parameter model will always agree with 
the actual fractions of games won, lost, and tied by each team after a balanced schedule has been completed. 
For the model with a single tie parameter, it is only the combination wins-plus-half-losses (or equivalent) 
which has this property. Thus, with team-specific tie parameters, there is nothing special about counting a 
tie as half a win; indeed, given any system awarding c^. points for outcome k, we can define a "round-robin 
points per game" , RRPPG, that is equivalent to RRWP if the point system is a linear transformation of 
ci = 1,C2 = 0, Co = \: 

2 

Ti = ^ CkRik- 

k=0 

After a round-robin tournament with this point system, the points obtained per game by each team will be 
equal to Ti. This implies that the Ti may sensibly be used to rank the teams after any collection of games. 

The world standard for soccer, for example, sets ci = 3, C2 = 0, co = 1 - that is, three points for a win 
and one for a tie. A model with a single tie parameter (or the idea, mentioned in Section 12.41 of counting 
ties literally as half a win and half a loss) is no longer reasonable here, since the teams could be ranked 
differently by Ti and by points even at the end of a balanced schedule (because the observed and expected 
proportions of ties for each team may not match). Instead, it is necessary to fit team-specific tie parameters 
and then rank the teams by Ti, since then observed points per game will equal Ti after a balanced schedule. 
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6 Examples 

6.1 Example 1 

Suppose there are four teams, a, 6, c, d. Teams a and b play twice, winning one game each, and c and d do 
the same. Team a plays c once, with a winning. Even though each team has at least one win and at least 
one defeat, it is seen that there is still a separation between the groups {a, 6} and {c, c?}, because the only 
inter-group comparison is a's win over c. 

Applying the procedure of Section 13.21 gives the relations a > b, b > a, c > d, d > c, a > c. Hence 
at>6, at>c, b l> a, c [> d, d\> c directly and a\> d, 6!>c, b ^ d indirectly, from computing the transitive 
closure. These are all the !> relationships, and so the teams are related as follows: 

a 

^ b 

> > c 

> > = d 

In other words, a and 6 as a group are "infinitely stronger" than c and c? as a group, and the maximum 
likelihood probability of a team in {a, 6} defeating a team in {c, d} is 1. 

The maximum likelihood estimates for the probabilities of a defeating b and for c defeating d are both 
(unsurprisingly) 0.5, and so the RRWP's for a, b, c and d are 0.833, 0.833, 0.167 and 0.167 respectively. 

6.2 Example 2 

Suppose four teams play the following games. The first-named team is at home in each case, and we will 
want to fit a home field advantage. 

• a vs 6: a wins 

• 6 vs a: 6 wins 

• c vs a: a wins 

• c vs d: c wins 

• d vs c: d wins 



19 



(This is in fact the same set of games as in Example 1.) 

The procedure of Section [3. 41 is concisely expressed in the table below. A number in a particular row and 
column denotes a > relationship between the items of that row and column; the number itself denotes the 
number of the step at which that [> relationship was added. An asterisk denotes that the !> relationship was 
added on the second iteration. A third iteration failed to add any more [> relationships. 





aH 


bH cH 


dH 


aV 


bV 


cV 


dV 


aH 


1 


6* 


6 


4 


2 


3* 


3* 


bH 




1 3 




2 


4 


3* 


3 


cH 




1 








4 


2 


dH 






1 






2 


4 


aV 




2 




1 




3* 


3 


bV 










1 


6 




cV 












1 




dV 














1 



The relationships between each 77-team and each V-tea.m are therefore (the <C relation between CH and 
AV of course means AV 3> CH) : 





aV 


bV 


cV 


dV 


RRWP 


aH 


> 


> 


> 


> 


0.750 


bH 


> 


> 


> 


> 


0.667 


cH 


< 




> 


> 


0.250 


dH 






> 


> 


0.333 



The RRWP's apply to each of teams a, b, c, d as a whole (averaged over the items representing their home 
and road performances, as in 

The estimated probabilities of the events that occurred are all 1 , and hence the maximum value of the 
likelihood is 1. This result contrasts with Example 1, which was based on the same data, but without a 
home field effect. The home field advantage in this Example is overwhelming, to the extent that the winner 
of a game between a and 6 is predicted "with certainty" to be the home team, but the difference in strength 
between a and c is so large that it even overcomes this home field advantage. Finally, since aV S> cH but 
bV ^ cH, the RRWP for team a is greater than that for team b, as shown. 
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6.3 Example 3 

Suppose now that three teams a, b, c play a round-robin in which each team meets the others four times, as 
follows: 

a vs. b: b wins once, 3 ties 

a vs. c: a wins 4 times 

b vs. c: c wins 2 times, 2 ties. 

The won-lost-tied records of these three teams are as follows: 

a: won 4, lost 1, tied 3 
b: won 1, lost 2, tied 5 
c: won 2, lost 4, tied 2 

With a single tie parameter, the algorithm of Section IT!^ shows that all items a+, b+, . . . , c— are related 
=: the likelihood is maximized by probabilities strictly between and 1. The estimated probabilities are: 



i 


3 




Pzj2 


PijO 


a 


b 


0.464 


0.126 


0.410 


a 


c 


0.513 


0.101 


0.385 


b 


c 


0.316 


0.229 


0.455 



and the round-robin wins, losses, and ties per game, and RRWP, are: 



Team, i 


Rii 


Ri2 


RiO 


R^ 


a 


0.489 


0.114 


0.398 


0.6875 


b 


0.221 


0.346 


0.432 


0.4375 


c 


0.164 


0.414 


0.420 


0.3750 



Team 6 is estimated to be stronger than team c, since ptci > Pbc2, which is not surprising considering 
that b has 7 points in the round-robin (2 points for a win and 1 for a tie), while c has only 6. Note that 
the RRWP agrees with the actual winning percentage for each of the three teams, but the round-robin wins, 
losses, and ties per game for each team predicted by the model do not agree with the actual numbers. 

If we apply the algorithm of Section f4.3l to fit team-specific tie parameters, we obtain the !> relationships 
shown in the two tables below. A number in a table indicates, as previously, that the teams or items in 
question are related by !>, with the number itself indicating the step at which the > was found. 
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a 


b 


c 


a 


1 


3 


2 


b 


2 


1 


3 


c 


3 


2 


1 





a+ 


b+ 


c+ 


a— 


6- 


c— 


a+ 


1 




4 




2 




b+ 


3 


1 


3 


2 


3 


2 


c+ 


3 




1 




2 




a— 


3 


4 


2 


1 


3 


4 


b- 


2 




4 




1 




c— 


3 


2 


3 


3 


3 


1 



This shows that a— c+ and c— ^ a+, with all other relevant relationships being (Only relationships 
of the form i vs. j, i+ vs. j~ with i ^ j concern games that can be played.) From the second Table of 
Section^21 the estimated probability of a tie between a and c is zero. The entire set of estimated probabilities 



i 


j 




Ptj2 


PijO 


a 


b 


0.210 


0.040 


0.750 


a 


c 


0.790 


0.210 





b 


c 


0.210 


0.290 


0.500 



and the RRWP values are: 



Team, i 



Rii 



R 



i2 



R 



iO 



0.500 0.125 0.375 0.6875 
0.125 0.250 0.625 0.4375 
0.250 0.500 0.250 0.3750 

Note now that team c is estimated to be stronger than 6, since its probability of winning against b is higher 
than its probability of losing. In other words, under the team-specific tie parameter model, the ranking of 
teams by strength no longer necessarily agrees with the ranking under a point system of 2 per win and 1 per 
tie. The explanation here is that b has almost no chance of defeating a, but gains a substantial number of 
points against a from ties - enough, in fact, to overcome its deficit of points from games against c. 

Note also that in this model the predicted proportions of round-robin wins, losses, and ties for each team 
agree with the observed proportions. 
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Team 


Abbr. 


Div. 


Oct 18 
W-L RRWP 


Oct 25 
W-L RRWP 


Final 
W-L RRWP 


Air Force 


AFA 


Mtn 


3-1 


.667 


4-1 


.635 


7-1 


.849 


San Diego State 


SDSU 


Pac 


3-0 


.600 


4-0 


.967 


7-1 


.821 


Brigham Young 


BYU 


Pac 


2-1 


.233 


3-1 


.488 


7-1 


.813 


Wyoming 


Wyo. 


Mtn 


3-0 


.967 


4-0 


.967 


6-2 


.764 


Rice 


Rice 


Mtn 


2-1 


.667 


2-2 


.450 


5-3 


.629 


Colorado State 


CSU 


Mtn 


3-1 


.667 


4-1 


.658 


5-3 


.629 


Utah 


Utah 


Pac 


2-1 


.600 


2-2 


.867 


5-3 


.613 


Fresno State 


FSU 


Pac 


1-2 


.300 


2-2 


.609 


5-3 


.576 


Texas Christian 


TCU 


Mtn 


2-1 


.667 


2-2 


.610 


4-4 


.560 


Southern Methodist 


SMU 


Mtn 


2-2 


.667 


3-2 


.529 


4-4 


.499 


Texas-El Paso 


UTEP 


Pac 


2-1 


.667 


2-2 


.441 


3-5 


.337 


San Jose State 


SJSU 


Pac 


2-1 


.667 


2-2 


.380 


3-5 


.337 


Tulsa 


Tulsa 


Mtn 


0-3 


.200 


0-4 


.100 


2-6 


.372 


New Mexico 


UNM 


Pac 


0-4 


.200 


1-4 


.133 


1-7 


.100 


Nevada-Las Vegas 


UNLV 


Mtn 


0-4 


.133 


0-5 


.100 


0-8 


.067 


Hawaii 


Haw. 


Pac 


0-4 


.100 


0-5 


.067 


0-8 


.033 



Table 3: Summary of results from WAC data of Section IfTH 




6.4 Example 4 

In Fall 1998, the Western Athletic Conference (WAC), an American collegiate sports association, had 16 
members. For (American) football competition, these were divided geographically into two divisions. Each 
team played all the other teams in its own division, but faced only one opponent from the other division. The 
teams, their abbreviated names and their division membership are shown in the first three columns of Table|3| 



"Mtn" denotes Mountain division, "Pac" Pacific. Game results were obtained from USA Today (1998) 

We focus on three subsets of the game results: the games played before October 18, those played before 
October 25, and all the games played in the regular season. There were no ties, and we have chosen not to 
model any home field advantage. Model was fitted to each subset of the data, and RRWP calculated for 
each team as in Section Table 13 summarizes the wins, losses and RRWP for each team and subset. 

The RRWP values do not show which relationships, =, 3>, ^ or ^, hold between each pair of teams. 
These are depicted in Figure ^ for each of the three subsets of the data. The relation =, as described in 
Section 13.21 divides the teams into equivalence classes; in this example, all the classes except one contain 
only one team (which is used to label the class), while the remaining class, labelled "Others", contains all 
the remaining teams. The vertical position of the classes on the page is determined by RRWP; an arrow 
connecting two classes denotes a 3> relationship between them. (For simplicity, arrows implied by transitivity 
are not shown; thus, for example, Wyo. ^ UNM in all three diagrams.) 

Most teams played only three games before October 18. As a result, there are numerous ^ and ^ 
relationships among the teams, and the relationship diagram, shown in Figure^, is complicated. However, 
the diagram explains two apparent anomalies in the RRWP values from Table 13 SDSU's three wins came 
against teams that lost all their games, explaining the low RRWP of .600, while BYU's low RRWP of .233 
comes from defeating two winless teams (Hawaii and UNLV) and losing to FSU, which was FSU's only win. 







Relationship 


Winner 


Loser 


Before 


After 


Air Force 


Tulsa 


> 


> 


Wyoming 


Rice 




> 


CSU 


TCU 






SMU 


UNLV 


> 


> 


SDSU 


Utah 




> 


BYU 


SJSU 


< 




FSU 


UTEP 


< 




UNM 


Hawaii 




> 



Table 4: Game results Oct 18-Oct 25 and effect on relationships 
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Each team played once between October 18 and October 25, with the results shown in Tabled] Results 
where the winning team was already related to the losing team by 3> or ^ do not affect the relationships, 
but a relationship of ^ changes to ^ and one of <C changes to =, pending the results of other games. Thus 
BYU and FSU join SJSU and UTEP in the "Others" class, while now SDSU > Utah and UNM > Hawaii. 
This simplifies the relationship diagram considerably, as shown in Figure Note that Utah now has the 
third-highest RRWP because the team's two losses came against the top two teams. 

At the end of the season, the relationship diagram was as shown in Figure Thirteen of the sixteen 
teams are now related so that estimated win probabilities for games between these teams are strictly 
between and 1, and can be estimated with standard techniques. UNM, UNLV and Hawaii are still related 
<C to the "Others" class, since none of these teams have any wins against teams in the "Others" class. 
Because UNM ^ Hawaii but UNLV ^ UNM and Hawaii (that is, UNM's only win of the season came 
against Hawaii, while UNLV played neither team), the three teams' RRWP's rank as shown in the last 
column of Table |31 

The final RRWPs for the teams in the "Others" class generally agree with the teams' won-lost records. 
Differences in RRWP between teams with the same numbers of wins arise for two reasons: the Mountain 
division is stronger overall than the Pacific, and some teams faced a stronger opponent in their inter-divisional 
game than others. For example. Air Force (Mountain) came out ahead of SDSU and BYU (Pacific), while 
SDSU was ahead of BYU because their inter-divisional opponents were Tulsa and UNLV respectively. On 
the other hand. Rice and CSU had identical 5-3 records and had interdivisional opponents with identical 3-5 
records, so both pairs of teams ended with identical RRWP's. 

7 Discussion 

In the Bradley- Terry model, we have shown that the separation properties of the data, and therefore the 
determination of the probabilities estimated to be or 1 by maximum likelihood, can be found by considering 
the transitive closure of certain graphs. This makes for straightforward computation, and avoids the need for 
the linear programming routines that are required for logistic regression in general. We also suggest that the 
Round-Robin Winning Percentage (RRWP) or Round-Robin Points Per Game (RRPPG) provide a sensible 
way of summarizing the results, because they are defined regardless of the separation properties of the data. 

One practical application of the RRWP (or RRPPG) is to provide "mid-season" rankings for a league 
of teams which will eventually play a balanced schedule, but have not yet completed it. At the end of the 
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season, the each team's RRWP wih be equal to the actual winning percentage used by the league to order 
the final standings, but when only part of the season has been played, the RRWP should be a more accurate 
measure of a team's performance to date, as it considers the strength of the completed portion of their 
schedule. Another application would be to a league which, because of considerations such as travel or length 
of season, does not play a balanced schedule. Traditionally, the teams in such leagues are ranked according 
to winning percentage, which benefits teams who happen to play weaker opponents more often. 

We also note that in models with team-specific order effects (such as (O or team-specific tie parameters 
(such as (jSJ), there is more than one parameter per team, so that ranking the teams by a single strength 
parameter is either impossible (in the former case) or not necessarily desirable (in the latter). RRWP and 
RRPPG provide a uniform means for ranking which may be preferable depending on the application. 

Aknowledgments 

We would like to thank G. Hatfield and R. Stagat for useful discussions of applications of the Bradley- Terry 
model. JTW acknowledges support by the Swiss Nationalfonds, and by the Tomalla Foundation, Zurich. 

References 

Albert, A. and J. A. Anderson (1984). On the existence of maximum likelihood estimators in logistic 
regression models. Biometrika 71, 1-10. 

Bradley, R. A. and M. E. Terry (1952). The rank analysis of incomplete block designs. I. The method of 
paired comparisons. Biometrika 39, 324-345. 

Butler, K. A. (1997). Some Problems in Paired Comparisons and Goodness of Fit for Logistic Regression. 
Ph. D. thesis, Simon Fraser University. 

Davidson, R. R. (1970). On extending the Bradley-Terry model to accommodate ties in paired-comparison 
experiments. Journal of the American Statistical Association 65, 317-328. 

Davidson, R. R. and R. J. Beaver (1977). On extending the Bradley- Terry model to incorporate within-pair 
order effects. Biometrics 33, 693-702. 

Davidson, R. R. and P. H. Farquhar (1976). A bibliography on the method of paired comparisons. Bio- 
metrics 32, 241-252. 



26 



Fienberg, S. E. (1979). Log linear representation for paired comparison models with ties and within-pair 
order effects. Biometrics 35, 479-481. 

Ford, L. R. (1957). Solution of a ranking problem from binary comparisons. Amer. Math. Monthly 64{8, 
part II), 28-33. 

Joe, H. (1990). Extended use of paired comparison models, with application to chess rankings. J. Roy. 

Statist. Soc. Ser. C 39{l), 85-93. 

Papadimitriou, C. H. and K. Steiglitz (1998). Combinatorial optimization: algorithms and complexity. 
Mineola, NY: Dover Publications Inc. Corrected reprint of the 1982 original. 

Santner, T. J. and D. E. Duffy (1986). A note on A. Albert and J. A. Anderson's conditions for the 
existence of maximum likelihood estimates in logistic regression models. Biometrika 73, 755-758. 

USA Today (1998). Newspaper. At World Wide Web page 

http : //www . usatoday . com/sports/f ootball/ sf c/wac/sf c j 98 .htm. 



27 



